Papers with Contrastive decoding
Explaining and Improving Contrastive Decoding by Extrapolating the Probabilities of a Huge and Hypothetical LM (2024.emnlp-main)
Copied to clipboard
| Challenge: | Contrastive decoding (CD) improves the next-token distribution of a large expert language model (LM) using a small amateur LM. |
| Approach: | They propose a new unsupervised decoding method called Asymptotic Probability Decoding (APD) that extrapolates the probability curves from the LMs of different sizes to infer the asymptototic probabilities from an infinitely large LM. |
| Outcome: | The proposed method improves the next-token distribution of a large expert language model using a small amateur LM. |
Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs’ Hallucinations (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate hallucinations include prompt engineering and model optimization, but lack domain generalization and potential errors in fine-tuning data may exacerbate the hallucism. |
| Approach: | They propose an expert-aware adaptive contrast decoding that uses expert differences in MoE’s higher layers to mitigate hallucinations on QA tasks. |
| Outcome: | The proposed method outperforms baseline models on four datasets Large language models (LLMs) show strong performance but suffer from hallucinations, limiting their application. |